Methods employing stored preference data to identify video of interest to a consumer

ABSTRACT

The picture information of video can be used in various ways to identify the video or its topical subject matter. This enables numerous novel arrangements in which particular video of interest to a particular consumer can be discerned. Preference data for the consumer, by which video of interest is identified, can be shared with third parties, e.g., in exchange for a fee. A variety of other embodiments and features are also detailed.

RELATED APPLICATION DATA

This application is a division of application Ser. No. 11/775,728, filedJul. 10, 2007 (now U.S. Pat. No. 7,697,718), which is a division ofapplication Ser. No. 10/172,734, filed Jun. 13, 2002 (now U.S. Pat. No.7,263,202), which claims priority benefit to provisional application60/303,173, filed Jul. 5, 2001.

Filed on the same day as patent application Ser. No. 10/172,734 were twocompanion applications, Ser. Nos. 10/172,735 and 10/172,733, which bothclaimed priority benefit to application 60/303,173. Filed on the sameday as application Ser. No. 11/775,728 was companion application Ser.No. 11/775,734, which was also a division of Ser. No. 10/172,734, withpriority benefit to application 60/303,173.

TECHNICAL FIELD

The present technology concerns use of consumer preference informationin video processing.

BACKGROUND

(Much of the disclosure from which the present claims are drawn is foundtowards the end of this specification.)

Digital watermarking is the science of encoding physical and electronicobjects with plural-bit digital data, in such a manner that the data isessentially hidden from human perception, yet can be recovered bycomputer analysis. In physical objects, the data may be encoded in theform of surface texturing, or printing. Such marking can be detectedfrom optical scan data, e.g., from a scanner or web cam. In electronicobjects (e.g., digital audio or imagery—including video), the data maybe encoded as slight variations in sample values. Or, if the object isrepresented in a so-called orthogonal domain (also termed“non-perceptual,” e.g., MPEG, DCT, wavelet, etc.), the data may beencoded as slight variations in quantization values or levels. Thepresent assignee's patents U.S. Pat. Nos. 6,122,403 and 6,408,082, andapplication Ser. No. 09/503,881, are illustrative of certainwatermarking technologies. Watermarking techniques are also taught inthe following Philips patents: U.S. Pat. Nos. 6,252,972, 6,209,092,6,198,832, 6,157,330, 6,131,161, 6,031,815, 5,940,134, 5,933,798, and5,873,022.

Watermarking can be used to tag objects with a persistent digitalidentifier, and as such finds myriad uses. Some are in the realm ofdevice control—e.g., tagging video data with a do-not-copy flag that isrespected by compliant video recorders. (The music industry's SecureDigital Music Initiative (SDMI), and the motion picture industry's CopyProtection Technical Working Group (CPTWG), are working to establishstandards relating to watermark usage for device control.) Otherwatermark applications are in the field of copyright communication,e.g., indicating that an audio track is the property of a particularcopyright holder.

Other watermark applications encode data that serves to associate anobject with a store of related data. For example, an image watermark maycontain an index value that serves to identify a database recordspecifying (a) the owner's name; (b) contact information; (c) licenseterms and conditions, (d) copyright date, (e) whether adult content isdepicted, etc., etc. (The present assignee's MarcCentre service providessuch functionality.) Related are so-called “connected content”applications, in which a watermark in one content object (e.g., aprinted magazine article) serves to link to a related content object(e.g., a web page devoted to the same topic). The watermark canliterally encode an electronic address of the related content object,but more typically encodes an index value that identifies a databaserecord containing that address information. Application Ser. No.09/571,422 details a number of connected-content applications andtechniques.

DETAILED DESCRIPTION

EPGs organize and present (e.g., by time or topic) upcoming videoprogram content. As cable and satellite services increasingly havehundreds of channels, such guides become essential. EPGs are detailed ina variety of patents, including many assigned to Gemstar and StarSight,including U.S. Pat. Nos. 6,216,265, 6,118,492, 6,133,909, 6,144,401,6,167,188, 6,247,176, 6,151,059.

In typical EPG systems, a service provider (e.g., TV Guide) collectsprogramming information from national, network and local programsources, and compiles it into a database. The database is indexed bygeography and delivery source (e.g., Portland Oreg.; AT&T Cable). Once auser's location and delivery source is known, the database can bequeried to identify programming that is upcoming, e.g., for the next twohours. This data is typically presented in tabular (grid) form on theuser's video screen. Advertising and other information may be presentedwith the EPG data on the screen.

EPG data is presently conveyed to the consumer for display on-screenthrough “out-of-band” techniques, such as the vertical blanking intervalin analog video. Some systems have dedicated channels through which suchdata is presented.

One drawback of such systems is their reliance on time-of-day as the keyby which program schedules are determined. If a sports game runs late,or if breaking news forces an interruption in normal programming, theEPG does not reflect the change.

In accordance with one aspect, watermark data identifying a program isdecoded from incoming video. This information is then checked againstprogram identification data in the EPG and, if a discrepancy is noted,then a change in the programming is indicated on the displayed EPG.Consider Superbowl Sunday. The EPG database may indicate that aSuperBowl broadcast on channel 2 is scheduled to end at 5:00 p.m. At5:15, a watermark is decoded from channel 2 and conveys an identifierassociated with the SuperBowl, indicating that the SuperBowl program hasrun past its originally-allotted time. The EPG can update its entry forthe Superbowl, extending its ending time to 5:15 (the current time),5:30 (the next half-hourly interval), or otherwise mark it as changed.Data for subsequent programming on that channel can likewise be treatedas changed (e.g., by pushing back all programming 15 minutes, or to thenext half-hourly interval, or otherwise), and displayed in the EPGaccordingly.

Continuing this scenario, assume the SuperBowl broadcast ends at 5:20.This ending time can be detected by failure to detect theSuperBowl-identifying watermark from program material on channel 2 for apredetermined interval, such as 5 minutes. At 5:30, a new watermark IDis detected—this one corresponding to the program originally scheduledfor broadcast at 5:00. In this case, the database can shift by 30minutes the expected times of the 5:00 program, to 5:30. Viewers whorefer to the EPG at 5:35 will thus be presented with a timetable thataccurately reflects the currently available programs.

The expected timing of future programs can also be shifted in the EPGdatabase and display, with the service provider deciding how far out intime to continue this shift. Certainly by midnight, for example, theprogramming would be expected to return to its originally scheduledtiming, with some compensating program change (e.g., an abbreviation ofthe evening news) making up for the SuperBowl overrun.

In some embodiments, the EPG interface presented to the user indicatesthat programming times for a particular channel are uncertain. In theexample just given, for example, a graphical display of a program gridfor channel 2 may be highlighted in yellow from 6:00 p.m. untilmidnight, indicating that a change has made precise air-times uncertain.As each half-hourly slot arrives, however, the detection of a watermarkfrom the video then being sent permits at least the currently-availableprogramming to be accurately displayed. (Of course, a variety of othertechniques can be used to indicate schedule uncertainty, other thanyellow highlighting. Distinctive text presentation within the grid(e.g., italics), special effects (e.g., flashing text or backgroundcolor), textual alert messages, and a great variety of otherarrangements can be used to alert the viewer to the uncertainty.)

In addition to indicating uncertainty in the program schedule, the EPGpresentation can also be altered to indicate that program times havebeen shifted from their expected values (i.e., certain times, butdifferent). Again, a variety of user interface techniques can be used tosignal this fact to viewers (including techniques detailed above).

Many VCRs, and essentially all Personal Video Recorders (PVRs), rely onEPG data to set start and stop times for recording functions. Thesedevices can respond to watermarks, and/or EPG information that isupdated by reference to watermarks, to capture the desiredprogram—regardless of program delays.

Assume that the 5:00 program following the SuperBowl on channel 2 isSeinfeld, and the user has earlier specified that Seinfeld (30 minutes)should be recorded. At 5:00, the recording commences. Soon, however, thewatermark data reveals that Seinfeld isn't yet playing. Out of abundanceof caution, the device may continue to record. Or it may stop after,e.g., a five minute grace period, and try again, re-starting at the nexthalf-hourly interval. Or, after stopping, it may immediately resumingrecording if a Seinfeld watermark is detected between 5:05 and 5:30.

If the device does not stop, but records continuously from 5:00 onward,it may continue until 30 minutes after a Seinfeld-identifying watermarkis first identified. Thus, if Seinfeld starts at 5:20, the device willcontinue to record until 5:50 (and optionally for a brief interval afterthat time). If a Seinfeld watermark isn't detected within apredetermined window, e.g., 90 minutes, of the expected air-time, therecording device may conclude that the Seinfeld broadcast has beencanceled (avoiding recording of hours of unwanted programming).

In other embodiments, the stop time for a recording isn't set byreference to EPG data, or by reference to a known interval (e.g., 30minutes) after a start time. Instead, the device stops only when awatermark identifying a desired program is no longer detected. (Here, aselsewhere, “no longer detected” typically requires absence of detectionfor a period of several minutes, to account for commercials and otherinterruptions that may not convey the watermark of the desired program.)

If the recording device began recording at 5:00, it captured part of theSuperbowl broadcast. Rather than save this unwanted program material, itmay be deleted. In PVRs, and other devices with digital storage, thestorage space allocated to the unwanted programming can simply be markedas un-used by the desired program, and returned to the pool of availablestorage. In one such embodiment, recorded programming is discarded untila time a predetermined interval (e.g., 90 seconds) before firstdetection of the Seinfeld-identifying watermark.

In other arrangements, instead of deleting the non-Seinfeld program, itmay be retained in storage. By reference to the updated EPG data, or thewatermark, the unwanted programming can be identified as the end of theSuperbowl. This information can be logged in the device's index ofrecorded material, together with an identification of its length, andoptionally the start and stop times of the original broadcast. (Otherinformation, such as the location of the Superbowl video data in thedevice's file structure can also be maintained, but such information istypically transparent to the user.) When a table of recorded contents ispresented to the user on-screen, the Superbowl excerpt can be includedamong the listings—possibly set-off by distinctive UI presentation toindicate that it was an unintended capture. When the user is firstpresented with this unintended program capture, the system may ask theuser whether it should be retained or deleted. The system may have adefault option, e.g., that unless the user acts to preserve theaccidentally-captured video, it is deleted.

In tape-based systems, if recording started at 5:00, and at 5:25 thewatermark corresponding to the recorded program still indicates anon-Seinfeld program, the tape may be automatically rewound to the pointwhere the 5:00 recording commenced. Then, at 5:30, recording cancommence anew, on the possibility that Seinfeld has been shifted to thenext half-hourly slot and will be properly captured by recording from5:30 to 6:00.

Apart from accurately presenting program information, and correctlyrecording desired programs, watermarking can be employed in connectioncontent security and authentication. Consider the delivery of Pay PerView (PPV) content. The PPV content may have usage rules associated withit. These rules may, e.g., disallow recording, or fast forwarding, orrewinding, or pausing, etc. These usage restrictions may be conveyed byout-of-band or virtual channels, such as data transmitted prior to thevideo program, or during the vertical blanking interval, or in packetheaders. In accordance with certain embodiments, watermarks are used torepresent this information.

For example, a video signal can include a watermark with an eight bitpayload. The first bit, if set to “1,” can indicate no copying. Thesecond bit, if set to “1,” can indicate one copy permitted. The thirdbit, if set, can indicate pausing is permitted. The fourth bit cancorrespond to rewind permission, the fifth to fast forward permission,the sixth can indicate that any copy must be made on a certain type ofmedium (e.g., a local PVR), the seventh can indicate that any copy mustbe stored in encrypted form, etc. (If copy-once is permitted, the videocan be modified during the permitted copying operation to signal thatno-more-copies are authorized. This may be done, e.g., by applying afurther watermark to the signal.)

In still other embodiments, watermarks can be used in bandwidthoptimization strategies to provide augmented information, orprogramming, to a consumer. Consider a consumer who is interested inmotorcycling. This preference may have been specified explicitly by theconsumer, or may have been inferred through his behavior (e.g., hisobserved history of linking to on-line resources relating tomotorcycles). The topic of motorcycling may correspond to a particular16 bit identifier in a subject matter index (allowing 64 thousandsubjects). During hours when the PVR is not being actively used, it mayscan through all channels looking for material that is coded with themotorcycle subject code (perhaps among several others). If such materialis encountered, it is recorded, and a corresponding entry is made in thePVR's local table of contents. When the consumer next uses the device,he can see that a program of potential interest has been recorded.

The same technology can be used with advertising. Advertising can betopically coded to identify the subject matter. If advertising isencountered having the motorcycle subject code, it can be captured andlocally stored for possible later display. In this case, thepresentation of the advertising can be requested by the user (as withany other recorded program), or the captured advertisement can beinserted in a commercial slot in other video programming (perhapssupplanting another commercial that is not as likely to capture theconsumer's interest).

Such encoding of video content with subject-indicating codes is limitedby factors such as lengths of the codes, robustness tocompression/decompression and other distortion, video degradation, andother factors. In one system, a watermark with an 80 bit payload can beencoded in video, permitting recovery of 5 different content codesevery, e.g., 30 seconds. The same content codes can be repeated every 30seconds. Or, by repeating them less frequently, more codes can beconveyed (e.g., codes 1-5 in time 0-30 seconds; codes 6-10 in time 30-60seconds, codes 1-5 in time 60-90 seconds, etc.). Of course, thiswatermark may be overlaid or interleaved together with other watermarksconveying other information.

Instead of conveying absolute subject matter codes, each video excerptcan convey a unique ID that is used to access associated meta data in adata store. The data store may be local (e.g., downloaded to a set-topbox periodically), or remote (e.g., at a cable head-end or elsewhere).Thus, a Seinfeld program may have a single code. But when that code isused to access a corresponding database record with meta data, therecord may reveal 10 subject matter codes (e.g., comedy, New York City,Jerry Seinfeld, motorcycling, episode 29, episode title, etc.).

Different subject codes (or unique IDs) can be used for differentportions of a video program. So a ninety second clip that relates to amotorcycle may be coded to identify this subject matter, withoutso-coding the remainder of a program. The different portions need not bejust temporal portions. Through object segmentation technology, such asis used in MPEG-4, different on-screen objects can be encoded withdifferent watermark identifiers. Thus, a motorcycle in that ninetysecond clip may be encoded with a watermark indicating its motorcyclesubject matter, while a BMW automobile in the same scene may be encodedwith a different watermark.

In monitoring program material for desired codes, the consumer devicecan have a buffer in which the previous 15 seconds of video is alwaysavailable. Thus, if a desired watermark is detected, video from 15seconds prior to the detection can be written to long-term storage—toaccount for possible latency in watermark detection.

In a typical scenario, there may be 100 subject codes for which aconsumer's PVR is watching, corresponding to 100 subjects of potentialinterest to the consumer. As each watermark is detected, it is checkedagainst this list and, if a match is found, the video is captured(including the buffered 15 seconds prior to detection) for laterdisplay. If the program material is watermarked with unique IDs insteadof literal subject matter codes, the consumer device can query adatabase for the corresponding subject matter codes, and record thecontent if a match with one of the 100 profiled subject matter codes isfound. In some embodiments the database is maintained remotely,introducing a delay as the decoded codes are sent to the data, and theresults relayed back. Other approaches can mitigate this delay. Forexample, some (or all) of the database can be cached at the consumerpremises. Another approach is for the consumer device to periodicallysend its 100 searched-for subject matter codes to the database, whichthen returns a list of the Unique ID records for which the consumerdevice should be on the lookout (i.e., those having the subject mattercodes that are searched for).

The watermark detection functions referenced above can take place atvarious different locations. In some embodiments, detection may takeplace at a device in the consumer home, such as in one or more of aset-top box, VCR, PVR, television monitor, etc. Information from suchdetection, in some embodiments, may be related back up the chain ofdistribution (e.g., a neighborhood distribution node, cable head-end,cable control center, national EPG database provider, etc.) Or thedetection can take place at any of the upstream locations. For example,a station through which all AT&T Cable signals destined for subscribersin Multnomah and Clackamas counties in Oregon can monitor all thosechannels.

In some applications, it is desirable to employ the watermark-basedsystems detailed above in connection with known prior art techniques.PPV usage data, for example, can be conveyed both in header dataassociated with encrypted content, as well as by watermark data. Onedata can be relied upon primarily (e.g., the header data) and, if absentor apparently corrupted, the watermark information can be relied uponinstead.

As should be evident from the foregoing, certain embodiments contemplatethat a unique watermark identifier is associated with each videoprogram. The identifier may be generic to a class of programs (e.g., allSeinfeld shows are identified by the same watermark), or each differentprogram may have a different ID. In the latter case, the watermarkpayload may have plural portions. One portion may ID a family ofprograms (e.g., Seinfeld episodes), and another portion may convey an IDuniquely identifying a particular program in that family (e.g., episode88, “The Big Salad”).

The watermark identifier can be used to access a corresponding databaserecord where information about the identified program is stored. It mayinclude the date the program was first broadcast (e.g., “Sep. 29,1994”), a synopsis of the program (e.g., “Jerry dates Newman's ex.George buys Elaine a big salad from the coffee shop, when his girlfriendtakes credit for buying it. George lets Elaine know that it was him thatbought that salad. Show ends with Kramer driving Gendison's white FordBronco down the interstate (ala OJ)”), rights-holder information,digital rights management information, copy control information, linksto related programs, links to related merchandise, links to on-lineresources including chat rooms and program archives, subject matterclassification codes, etc. This database may be stored remotely from theuser station, and remotely accessed by the user as necessary. Or some orall of the database and contents can be kept (or mirrored) in a datastore at the user's premises (e.g., in a set top box). Such a databasecan be configured in accordance with a user profile, e.g., specifyingthe class of programs to be detailed in the local database. Or theremote database can provide the local database with informationcorresponding to watermark IDs expected to be encountered in the next 7days (e.g., during early morning hours when system traffic is otherwiselow). A local user database can include information provided by the userand not shared with a remote database, including private notes aboutlikes/dislikes, etc. Or information provided by the user (includingdemographics and viewing habits) can be passed to a remote database.Such personal information in the remote database can be shared withother users, with a cable system operator, with the provider of theannotated database record, etc. The user may receive a fee in somecircumstances for sharing such information.

To provide a comprehensive disclosure without unduly lengthening thisspecification, the patents and applications cited above are incorporatedherein by reference.

Having described and illustrated the subject technologies with referenceto illustrative embodiments, it should be recognized that the technologyis not so limited. For example, it will be recognized that the conceptsdetailed above can be implemented with various forms of watermarkingtechnologies, and can be advantageously combined in straight-forwardfashion with other content-delivery systems. Moreover, it will berecognized that wherever in the prior art that vertical blankinginterval-based data communications techniques were used, the sameapplications may be served instead by conveying such data throughin-band video watermarking. Further, it should be recognized that theparticular combinations of elements and features in the above-detailedembodiments are exemplary only; the interchanging and substitution ofthese teachings with other teachings in this and theincorporated-by-reference materials are also contemplated.

1. A method comprising the acts: storing consumer video profileinformation; by reference to said stored consumer profile information,automatically identifying video of potential interest to said consumer;and in exchange for consideration provided to said consumer, providingat least some of said consumer video profile information to a thirdparty.
 2. The method of claim 1 in which said consumer video profileinformation includes consumer video preference data and consumer viewinghabit data.
 3. The method of claim 2 that includes, in exchange for saidconsideration, providing at least some of said consumer viewing habitdata to the third party.
 4. The method of claim 2 wherein the consumervideo profile information includes data about consumer video dislikes.5. The method of claim 1 wherein the storing comprises storing consumervideo profile information at a device at the consumer's premises,